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1. INTRODUCTION 

Optimization is a process for finding the most curtail value for given functions within a specific 
domain, this process is mostly studied and used in mathematics, computer and physics science, the principal 
part of the unconstrained optimization is minimizing an objective function that rely on real variable, with no 
restrictions on the values. Mathematically, let x E R” be real vector with n >1 component and let 
f : R” > R be smooth function, we consider below unconstrained optimization problem: 


min f (x) Vx E€ R” (1) 
Where the function f: R” —> R and its gradient g;, also conjugate gradient methods (C-G) have played special 
roles in solving large dimension nonlinear problems. C-G method is an iterative scheme that usually 
generated a sequence {x;} of an approximated solution for (1), [1]-[4] usually we use the repetition: 


Xi+1 = Xi + Pipi i = 0,1,2, on (2) 


Where ~; > 0 is a step size evaluated by line searches (Goldstein condition, wolfe condition, sufficient 
decent condition, and curvature condition) and the search direction p; is designed by: 


Pisa = —Gi+r + Bi Pi (3) 
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Where g; = Vf (x;) and f; is most influential parameter in C-G method. The different choices for the 
parameter f; correspond to different C-G method, and most famous f; check by the following scientists, 
Hestenes and Stiefel (H-S) [5], Polak-Ribiere-Polyak (P-R) [6], Fletcher and Reeves (F-R) [7], 
Alaa and Banaz [8], two formulas for 6; as shown in (4) and (5): 


P-R Tai 
Post) = —Gisi + ( aa ) Pi (4) 
i Ji 
new Ilir Ths Gi+1 A 
5 = <ü + (eS L [ass] z 5 
P41) Ji+ı + ( piyi H OF 9: ) Pi (5) 


Where yi = Ji+ı — gi and Si = Pipi = Xi41 — Xi. Furthermore, the design of CG-techniques had been 
studied by many researchers; for more details see [9]—-[14]. 

In the convergent condition and implementation of C-G method [15]-[17], one often requires the 
exact and inexact line search such as the wolfe conditions or the strong wolfe conditions. The wolfe line 
search is to find ø; such that: 


f (xi + pipi) < f (xi) + Oig Pi (6) 

Pi 9 (Xi + Pipi) = ogi Pi (7) 
With 0 <@<o. The strong Wolfe line search is to find @; such that: 

f (xi + pipi) < f) + 9:97 Pi (8) 

lpi 9 (xi + Givi)| S ogi Pi (9) 
Where 0 < 6<o <1 are constants [18]-[21]. 

In this paper, will present our new C-G training algorithms and their derivations in section 2. Global 
convergence condition and descent condition of our modified methods are proved in section 3. Numerical out 


comes for two new proposed algorithms are discussed in section 4. Finally, we conclude our methos and 
gives in section 5. 


2. DERIVATIVE OF DOUBLE MODIFICATION ON P-R METHOD 
2.1. Derivative of first proposed C-G method (g/?-*) 
Consider we have these new approaches: 


2 
yi = yi + y; where p = Mail 
Si Yi IIpill 


And our search direction with new parameter will be of the form: 


Diss = —(u + 1)gi+ı +8" p; where u € (0,1) (10) 


By changing y; in standard parameter (P-R) that in (4) by y; we have: 


2 
lail 
simya K L 
NP_R Pill SiYi 
fi ~ = G! 9: See aD 
l 
So, we get this new form: 
2 
lail 
abar n g 
NPR _ pP_R Pill SiYi 
E tr (12) 
l 


Now, we will show the steps of solving unconstrained optimization problems by the algorithms (2.2) and (2.3). 
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2.2. Algorithm of the first new method BE) 
In this subsection, the outlines of the first new method for solving unconstrained optimization 
problems is stated. 
Step 1 : begin with initial point x) € R”. 
Step 2 : set initial search direction pọ = —go, i = 0. 
Step 3 : if |lg;|| = 0 then stop, otherwise go to step 4. 
Step 4 : find the step size g; by using minimize function f (x; + pipi). 
Step 5 : set Xj44 = Xi + Didi. 
Step 6 : determine g;,1, if ||gi+1|| < 107° stop, else go to step 7. 
Step 7 : compute new search direction pi+ı by (10). 


Tg; 
Step 8 : if Ilgi ll? < lat smal is satisfied go to step 3, 


else i = i + 1 and go to step 4. 


2.3. Derivative of second proposed C-G method (B1"?-*) 
Consider our new search direction is of the form: 


pirt = —(1 + 0) gies tp E p (13) 
The logistic mapping form [1] and standard parameter in (4) gives: 
nk = u BP F(A — opi”) 
Where: 


w = 25 9 € 0,1) ,pe(0,1] D 


This implies that, 


PUDER Ih+1Yn (1 _ Gi+1 Si Inn) (15) 


SH Ron piyi gign 


2.4. Algorithm of the second new method (B)"?-*) 


t 
In this subsection, the outlines of the second new method for solving unconstrained optimization 


problems is stated. The steps of this algorithm are the same steps that in the first new outline (2.2). The only 
changeable step is step 7, we will compute the new parameter that in (15). 


3. DESCENT PROPERTY AND GLOBAL CONVERGENCE CONDITION FOR TWO NEW METHOD 
Assumption (A) [22]-[24]. 

3.1. The level set 6 = {x |f (x) < f(x;)} at x; is bounded, there exists a constant a > 0 such that: 
|x|] < a, Vx E€ 6 (16) 


3.2. If the function f is continuously differentiable and some neighborhood N of ô, and its g;(x) is Lipschitz 
continuous with Lipschitz constant L > 0, i.e., 


lav) -g@I < Lllr—-tllvr,t€6 (17) 
3.3. From two assumptions that in (16) and (17), that there exists a positive constant c such that: 

lIg(c)ll < c Yx € 8 (18) 
If function f is strongly convex, then there exists a constant u > 0 such that: 

ull- yl? < Wf @) - Vf) (x — y), forallx,y € 6 (19) 


3.4. Consider assumption (16) is hold. Suggest that (1) and (2), where p; is a descent direction and 9; 
satisfies the standard wolfe line search. If: 
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1 


diet paz = 00 (20) 
Then, lim inf ||g;|| = 0 
1-00 


Theorem (3.1): proof that the search direction pi}, with modified parameter PNPP of C-G method 
is descent direction g/,1P;41 < 0. consider the sequence { x;} is in the (2) where the step length œ; defined 
in two cases, exact line search and inexact line search. Then: 


Pisa = —( +1) Jiz + BRP? pi 


Multiple both sides of equation by g/,, and from (12), we get: 


2 
atari Ta ) 
P_R Pill SiYi 
Ji+1Pi+i = —( + 1) Ji+19i+1 + Pi” Gis Pi + Sa — Gis Pi (21) 
L 


Then we have two cases for selecting step size [11], so the first one by (ELS) that require of g/,, pi = 0, and 


: . 2. or 
the maintain term || Gi+1l| is positive, so: 


2 
GisrPi+1 =-(ut+1) lgl] (22) 


Thus (22) satisfies descent condition. 

But the second case of gi by ILS, it consists of g/,, pi # 0, in the above form we have Be so it 
satisfies the descent condition. By using strong wolfe condition in (21) 029! pi < 9)41 Pi < —O2g} pi where 
dz E (0,1) we get: 


R 


lai 
2 
lleill sP yi 
algi 


T 
Ji+1Yi( 


2 
GissDi+1 <-—(u+ 1)|I9i+41| = ogi Pi 


In this case if the wolfe condition satisfies, we can say that g/,, pi < p7 yi: 
2 
laill 


2 
lIpill sly; Pria 
gl gi A 


Tir vil 
7 2 
Ji+1Pi+ı S —Cut 1)|I9i+41| = 


We know that s; = pipi and y; = gi+1 — gi, so the equation will be of: 


2 
; daone 
eillpi 
GissPi+1 <-—(u + 1)|I9i+11| = —— 
llgill 
Then, 
oala 
lIgiesll?+97,491( j IP 
2 eillpi 
JL1Pi+ı < — (u + 1)|I9:+11| = —— 
llgill 
Xi 2 
By powell condition -|g/,,g;|<-0.2 Iisa ; 
lail 
elas 
IIgs lI2—0.2 |lgigal| (>) 
oillp:ll 


2 
IhaPi+ <-(u+ 1)|I9i+11| = laal 
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2 2 
T o ) sso (lep) 
eilleill |. Late = (u + 1) + oillpill 


2 
Jix1Pis1 S —|lgi l| [u +1)+ | 
ligal ligill 


Where pe(0,1) and @;, 02, llgill and FAR are positive terms, so T > 0, hence we get descent condition. 
Theorem (3.2): consider the assumption (A) is hold. Then the new pr P-R of the form (10) and (12), where Pi 


is checked by wolfe line search (7), then it satisfies global convergency condition lim inf ||g;|| = 0. 
l-00 
2 
lail 
atd le) 
Proof: pish = = — Gini +t Cts 7 ——*) Di 
Ji Gi 
2 
lail 
oli i i 
pil sty: 
[NES + Dllgeesll + |] 2] pa + |] eT pa (23) 
l 


Since ||g741Ynll < [lgil|llynl]l]- And by (LC) we have||yp|| < LII sill: 


L |lgi+al| afe) 
—; — i l 


EW 
A | = (u + DIlgi+ll + iz i ligil? 


Hence, we know that P! Gis1 < pyi , by using (7) and the form (p; = —gj). 
We get this form ||g;||? > = D; yi: 


Tije 
area 2) 
pias a; 
~~ || pill (25) 


PMA || < + De + yi 


pill + 


aL c apt Il 
Tyi 


Since p7 y; > 0 —— me , use this form in (25): 


girL c |lsi 


Ty, 
vente Peat 
Il Pil] Pid; di 
E Il pill + ——— =] pill 


pia” || s G+ De + Sis 


(26) 
After some algebraic operation in (26) we get this simple form: 


ee pe 
aT a pi TA 


Let ie |- = |lx1 — ell. y = max{| lxi — xili} V Xip X; E R 
oL oL 


SAE S DHE + oy) =r 


e Biz az lp E a > Dies = 
Pitt 


> Vici Gav? 
anew 


Finally, by using Lemma we get lim inf|lg;41|] = 0. 
X-00 
Theorem (3.3): assume that the sequence {x;} is generated by (2), then the search direction in (13) 
and new parameter B/™?~® satisfy the descent property, i.e., 


GisrPis1 S = (27) 
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Proof: multiply both sides of equation by gi+1, to obtain: 


NEW _ 


aap = -gagina tu Se — Tii Sten) gf p, (28) 


pp Vi 


If the search direction is chosen by (ELS), then g},, pi = 0 , so it satisfies (27), however, if the 
search direction (15) is inexact (i.e.,) Grid; + 0.We concludes that the first two terms of (26) are satisfies 


NEW _ Inn T Hridi (GesaYny2 


the descent condition i.e., gipM” = —Ilgieill? +u rage EEA ety Mata Gix1 Pi because the 


PRP method satisfies the (27) condition, so we have: 


NEW _ 


Jisa Si Gnas y: 
GisiPivt = —Ilgisill*? — ae ny? 


T 
PHY; gh Ton Gi+1 Pi 


Therefore, by using the form S; = @;p;, we get: 


Siit Pi” Iit 
Japi = -lgo -EEH Caper? 
Clearly, 6;, (gi+1 Pi)”, dT yi and (2g)? are positive term. Because of that it satisfies (27). 


Theorem (3.4): consider he asian (A) is hold. Then the new pe ZR of the form (13) and (15), 


where g; is checked by wolfe condition, then it satisfies lim inf ||g;|| = 0. 
l> œo 
. pNEW — Tn+1Yn Jita Si Int1Yn 
Proof: pea” = -gis1 +U (i- ) Pi 


dl gn Piyi IiIn 


GhesYn Gi+1 Si Ga) 


ill + 
all + fee ( 


IPERI < lgisall + u || 2) | II pill (29) 


Since |gn+1nlS |19i+1I[IIynll- And by (LC) we have|lyn|| < LII sill 


2 
EWI] < i ae A 4 y Lials (ree 
IAA | laill u gill? Il pill u lle? vill llgill2 Il ill 
EW sled c IIsill erly 
<ct+ || + i 30 
lesa” || H Tol izi H ipy] lg: ll pill (30) 


Hence, we know that p? gi+1 < pyi, by using (7) and form p; = —g;. We get this form ||g;||? = = pT Yi: 


choli ll c lisill f Łøcllsill 
IPS < c + p EERI pa + Sb (Heat) GD 
U 


Since p} y; => 0 a 


c Lo|lsill c Iisll (Locllsill\? 
PMI | ctus 7 Ilpill +e (ee) II pill 
[pMaY || s cuLEG + = 0° (32) 


1 
>er pNEW NEWI = 2 Misia = 
Pirs z 


wa NEW 
Perr 


By using Lemma (4.1), we get: 


lim inf || 9i+1\| = 0. 
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4. NUMERICAL RESULTS AND COMPARISONS 

Here, we show some numerical experiments of the new formulas in conjugate gradient method. 
Each problem has been tested with different values of n ranged from 4 to 5,000, where n is a number of 
variables of each test function [25]. We did our implementations in the FORTRAN 95 language with given 
initial points. Table 1 shows comparison tests consist of well-known non-linear functions with 8 different 
functions for each new method. For restart we use the powell condition. On the other hand, the comparative 
results illustrate in Table 2 which contain number of iteration (NI) and number of function (NF), the 
experimental results verify that the new algorithms are superior to standard (P-R), in general the rate of 
improvement of the first new method BN?~® is 3.238% in (NI) and 11.7118% in (NF), also for the second 
method £/™?-® that applied in logistic mapping is 40.556% in (NI) and 25.6373% in (NF), in general the 
magical improvement between (NI) and (NF) are 66.1933%. 


Table 1. Shows the comparison between standard P-R method and two modified methos BN?7* and BEM? 


Test function Recip Test function Box 
n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 9 9 9 9 NI (P-R) 22 22; 23 30 
NI (NP-R) 7 8 8 8 NI(LMP_R) 17 19 22 22 
NF (P-R) 28 28 28 28 NF (P-R) 159 159 171 270 
NF (NP-R) 19 23 23 23 NF(LMP_R) 103 130 171 174 
Test function OSP Test function Wolf 
n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 9 52 179 340 NI (P-R) 11 49 64 99 
NI (NP-R) 9 48 129 275 NI (LMP_R) 1l 43 48 99 
NF (P-R) 52 173 570 114 NF (P-R) 24 99 129 214 
NF (NP-R) 51 161 391 824 NF(LMP_R) 24 87 97 214 
Test function Wood Test function Miele 
n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 38 39 39 44 NI (P-R) 37 44 50 50 
NI (NP-R) 30 30 32 32 NI(LMP_R) 41 57 58 59 
NF (P-R) 83 85 85 95 NF (P-R) 116 148 180 180 
NF (NP-R) 67 68 72 72 NF(LMP_R) 104 142 144 146 
Test function Rozen Test function Wood 
n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 18 19 21 21 NI (P-R) 29 30 30 30 
NI (NP-R) 18 18 21 21 NI (LMP_R) 28 28 28 29 
NF (P-R) 46 49 55 55 NF (P-R) 67 69 69 69 
NF (NP-R) 46 46 53 53 NF(LMP_R) 63 63 63 64 
Test function cubic Test function Sum 
n 4 100 1000 5000 N 4 100 1000 5000 
NI (P-R) 29 31 31 31 NI (P-R) 3 14 23 31 
NI (NP-R) 25 25 25 26 NI(LMP_R) 3 14 20 28 
NF (P-R) 79 84 84 84 NF (P-R) 11 80 127 145 
NF (NP-R) 73 73 73 75 NF(LMP_R) 11 71 102 109 
Test function Powell 3 Test function Cubic 
n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 27 48 63 137 NI (P-R) 20 21 21 21 
NI (NP-R) 18 44 48 103 NI(LMP_R) 19 19 19 19 
NF (P-R) 55 97 127 289 NF (P-R) 55 59 59 59 
NF (NP-R) 37 89 96 230 NF(LMP_R) 54 54 54 54 
Test function Miele Test function Rosen 
n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 75 115 145 181 NI (P-R) 15 15 15 15 
NI (NP-R) 54 56 64 78 NI (LMP_R) 12 12 12 12 
NF (P-R) 185 295 382 480 NF(P-R) 41 41 41 41 
NF (NP-R) 128 144 165 205 NF(LMP_R) 32 32 32 32 
Test function Powell Test function OSP 

n 4 100 1000 5000 n 4 100 1000 5000 
NI (P-R) 98 124 138 138 NI (P-R) 8 49 161 276 
NI (NP-R) 17 17 22 31 NI(LMP_R) 8 47 156 276 
NF (P-R) 128 283 319 319 NF (P-R) 45 176 493 844 
NF (NP-R) 38 38 64 97 NF(LMP_R) 45 146 475 828 
Total of NI (P-R) 1328 Total of NI (P-R) 2266 
Total of NI (NP-R) 1285 Total of NI (LMP-R) 1347 
Total of NF (P-R) 4440 Total of NF (P-R) 4864 
Total of NF (NP-R) 3920 Total of NF (LMPR) 3617 
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Table 2. The percentage of improvement between the new methods BN?~* and B/“?~® with PRP method 
Tools Standard P-R (%) New BNP"? New BEMP-R 
NI 100 96.762 59.4439 
NF 100 88.2882 74.3626 


5. CONCLUSION 

We have shown two new powerful and superior non-linear C-G methods for solving unconstrained 
problems. Consider the standard approach proposed by P-R, we modified P-R two times, so on the one hand 
the first modified method delightful have double properties that performance well in numerical outcomes 
with compare to standard P-R and satisfies the descent and global convergent condition. On the other hand, 
the second modified method we applied in the form of logistic mapping and a new search direction, because 
of that it satisfies the global convergent and descent condition, farther more in numerical outcomes illustrate 
their magical performance with measured to standard P-R method. 
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